Skip to content

Stop failing reschedule-mode sensors on a stale executor success - #71194

Draft
rjgoyln wants to merge 1 commit into
apache:mainfrom
rjgoyln:fix-sensor-reschedule-stale-executor-event
Draft

Stop failing reschedule-mode sensors on a stale executor success#71194
rjgoyln wants to merge 1 commit into
apache:mainfrom
rjgoyln:fix-sensor-reschedule-stale-executor-event

Conversation

@rjgoyln

@rjgoyln rjgoyln commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

A sensor running with mode="reschedule" is intermittently marked failed when the executor success for one poke reaches the scheduler after the task instance has already gone back to scheduled (or queued) for the next poke. The scheduler reads the late event as a task that finished without recording its own state, and fails it:

Executor CeleryExecutor(parallelism=512) reported that the task instance
<TaskInstance: <dag_id>.<sensor_task_id> ... [scheduled] ti_id=...>
finished with state success, but the task instance's state attribute is scheduled.

The existing guards for this race all assume the defer path. A reschedule exit leaves next_method unset and keeps the same try_number, so the executor key still matches and none of them apply. A reschedule row for the current try is the signal that separates the two cases: it says the worker exited up_for_reschedule rather than being killed. Anything with no such row still takes the externally-killed branch.

It is resolved once per event batch alongside the bulk task-instance fetch, so the per-event path gains no query.

closes: #71172


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

A sensor running with mode="reschedule" was intermittently marked failed
when the executor success for one poke reached the scheduler after the
task instance had already been put back for the next poke. The existing
guards for this race only cover the defer path: a reschedule exit leaves
next_method unset and does not bump try_number, so none of them applied
and the event was treated as an external kill.
@boring-cyborg boring-cyborg Bot added the area:Scheduler including HA (high availability) scheduler label Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race condition between scheduler processing events and sensor reschedule

1 participant